Papers with Quality Estimation
Unsupervised Quality Estimation for Neural Machine Translation (2020.tacl-1)
Copied to clipboard
Marina Fomicheva, Shuo Sun, Lisa Yankovskaya, Frédéric Blain, Francisco Guzmán, Mark Fishel, Nikolaos Aletras, Vishrav Chaudhary, Lucia Specia
| Challenge: | Existing approaches require large amounts of expert annotated data, computation, and time for training. |
| Approach: | They propose an unsupervised approach to QE where no training is required . they use a dataset that enables work on both black-box and glass-box approaches . |
| Outcome: | The proposed approach rivals state-of-the-art supervised QE models in terms of correlation with human judgments of quality. |
deepQuest-py: Large and Distilled Models for Quality Estimation (2021.emnlp-demo)
Copied to clipboard
Fernando Alva-Manchego, Abiola Obamuyide, Amit Gajbhiye, Frédéric Blain, Marina Fomicheva, Lucia Specia
| Challenge: | Quality Estimation (QE) is a tool for machine translation that predicts how good translations are without comparing them to gold-standard references. |
| Approach: | They introduce a framework for training and evaluation of large and light-weight models for Quality Estimation (QE) they use pre-trained Transformers to train large and efficient QE models. |
| Outcome: | The framework provides access to state-of-the-art models based on pre-trained Transformers for sentence-level and word-level QE and a web interface for testing and visualising their predictions. |
Better Quality Estimation for Low Resource Corpus Mining (2022.findings-acl)
Copied to clipboard
| Challenge: | State-of-the-art Quality Estimation models lack robustness to out-of domain examples. |
| Approach: | They propose a method that uses multitask training, data augmentation and contrastive learning to achieve better and more robust QE performance. |
| Outcome: | The proposed method improves QE performance significantly in the MLQE challenge and the robustness of QE models when tested in the Parallel Corpus Mining setup. |
Multimodal Quality Estimation for Machine Translation (2020.acl-main)
Copied to clipboard
| Challenge: | Existing work has only explored textual context. |
| Approach: | They propose to use visual and text modalities to explore Quality Estimation for Machine Translation and integrate them into multimodal QE frameworks. |
| Outcome: | The proposed approaches improve on sentence-level and document-level predictions using visual features extracted from images. |
Rethinking the Word-level Quality Estimation for Machine Translation from Human Judgement (2023.findings-acl)
Copied to clipboard
| Challenge: | Word-level Quality Estimation (QE) of Machine Translation aims to detect potential translation errors in the translated sentence without reference. |
| Approach: | They propose to use a human-generated translation judgment to generate a word-level quality estimate (QE) using a translation error rate toolkit to detect translation errors without reference. |
| Outcome: | The proposed dataset is more consistent with human judgment and confirms the effectiveness of the proposed tag-correcting strategies. |
Assessing Quality Estimation Models for Sentence-Level Prediction (C18-1)
Copied to clipboard
| Challenge: | Using a relevant QE model is also very important in QE. |
| Approach: | They evaluate a wide range of advanced sentence-level Quality Estimation models including Support Vector Regression, Ride Regression and Bayesian Neural Networks. |
| Outcome: | The proposed models behave differently in evaluation settings depending on whether test data come from the same domain as the training data or not. |
Unveiling the Power of Source: Source-based Minimum Bayes Risk Decoding for Neural Machine Translation (2025.acl-long)
Copied to clipboard
| Challenge: | Maximum a posteriori decoding aims to maximize the estimated posterior probability, but high estimated probability does not always lead to high translation quality. |
| Approach: | They propose a method that seeks hypotheses with the highest expected utility by using quasi-sources as “support hypothese . they propose sMBR decoding which utilizes a reference-free quality estimation metric as the utility function. |
| Outcome: | The proposed approach outperforms QE reranking and the standard MBR decoding. |
Are Generative Models Underconfident? Better Quality Estimation with Boosted Model Probability (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have shown that text-generation models can be overconfident when there are multiple correct options. |
| Approach: | They propose a QE approach called BoostedProb which boosts the model’s confidence in cases where there are multiple viable output options. |
| Outcome: | The proposed approach achieves on average +0.194 improvement in Pearson correlation to ground-truth quality and outperforms more costly approaches like supervised or ensemble-based QE in certain settings. |
Quality Estimation for Image Captions Based on Large-scale Human Evaluations (2021.naacl-main)
Copied to clipboard
| Challenge: | a problem with automatic image captioning is that it produces low quality captions when used in the wild. |
| Approach: | They propose to model caption quality from a human perspective and *without* access to ground-truth references. |
| Outcome: | The proposed model can detect and filter out low-quality captions on previously unseen images. |
deepQuest: A Framework for Neural-based Quality Estimation (C18-1)
Copied to clipboard
| Challenge: | Predicting Machine Translation (MT) quality has been limited to word and sentence-level prediction. |
| Approach: | They propose a framework that can generalize neural QE approaches to the level of documents. |
| Outcome: | The proposed framework outperforms state-of-the-art approaches on document-level quality estimates and is 40 times faster to train. |
Translation Error Detection as Rationale Extraction (2022.findings-acl)
Copied to clipboard
| Challenge: | Recent Quality Estimation models rely on translation errors to predict overall sentence quality, but detecting specific errors is a more challenging task. |
| Approach: | They propose to use a semi-supervised method to detect translation errors by attribution of relevance scores to inputs to explain model predictions. |
| Outcome: | The proposed method can detect translation errors and is compared with human models using a set of feature attribution methods. |
Alleviating Distribution Shift in Synthetic Data for Machine Translation Quality Estimation (2025.acl-long)
Copied to clipboard
| Challenge: | Quality Estimation (QE) models evaluate the quality of machine translations without reference translations, serving as the reward models for the translation task. |
| Approach: | They propose a framework for alleviating distribution shift in synthetic QE data . they employ a constrained beam search algorithm and distinct generation models to enhance translation diversity. |
| Outcome: | The proposed framework outperforms SOTA baselines like CometKiwi in supervised and unsupervised settings. |
Beyond Glass-Box Features: Uncertainty Quantification Enhanced Quality Estimation for Neural Machine Translation (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Quality Estimation (QE) is an essential role in applications of Machine Translation (MT). |
| Approach: | They propose to fuse uncertainty quantification into a pre-trained cross-lingual language model to predict the translation quality. |
| Outcome: | The proposed method achieves state-of-the-art on the datasets of WMT 2020 QE shared task. |
Knowledge Distillation for Quality Estimation (2021.findings-acl)
Copied to clipboard
Amit Gajbhiye, Marina Fomicheva, Fernando Alva-Manchego, Frédéric Blain, Abiola Obamuyide, Nikolaos Aletras, Lucia Specia
| Challenge: | Recent success in Quality Estimation stems from the use of multilingual pre-trained models, where large models lead to impressive results. |
| Approach: | They propose to transfer knowledge from a strong QE teacher model to a much smaller model with a different, shallower architecture. |
| Outcome: | The proposed model performs better than distilled models with 8x fewer parameters. |
M2PO: Multi-Perspective Multi-Pair Preference Optimization for Machine Translation (2026.acl-long)
Copied to clipboard
Hao Wang, Linlong Xu, Heng Liu, Yangyang Liu, Xiaohu Zhao, Bo Zeng, Liangying Shao, Yichen Dong, Xinwei Wu, Jiang Zhou, Tianyu Dong, Xiangxiang Zeng, Longyue Wang, Weihua Luo
| Challenge: | prevailing methods for machine translation are often hindered by misleading reward signals. |
| Approach: | They propose a framework that aligns large language models to human preferences . they propose 'M2PO' to correct the bias towards partial errors . |
| Outcome: | The proposed framework outperforms open-source models and achieves parity with proprietary models. |
A Multi-task Learning Framework for Quality Estimation (2023.findings-acl)
Copied to clipboard
Sourabh Deoghare, Paramveer Choudhary, Diptesh Kanojia, Tharindu Ranasinghe, Pushpak Bhattacharyya, Constantin Orăsan
| Challenge: | Conventional approaches to QE involve training separate models at different levels of granularity viz., word-level, sentence-level and document-level . |
| Approach: | They propose to train a single model for sentence-level and word-level QE tasks in a multi-task learning framework and compare them to baseline models. |
| Outcome: | The proposed model improves on the single-pair, multi-patch, and zero-shot settings. |
Together We Can: Multilingual Automatic Post-Editing for Low-Resource Languages (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies on multilingual automatic post-editing systems for low-resource Indo-Aryan languages have focused on different models for different language pairs. |
| Approach: | They propose to use a multilingual automatic post-editing system to improve machine translations for low-resource Indo-Aryan languages. |
| Outcome: | The proposed model outperforms English-Hindi and English-Marathi models by 2.5 and 2.39 TER points. |
BERTology for Machine Translation: What BERT Knows about Linguistic Difficulties for Translation (2022.lrec-1)
Copied to clipboard
| Challenge: | Pre-trained transformer-based models have shown excellent performance in most benchmark tests, but lack a good understanding of the linguistic knowledge of BERT in Neural Machine Translation (NMT). |
| Approach: | They propose to use QE models to analyze BERT's syntactic dependencies and their impact on machine translation quality. |
| Outcome: | The proposed model is able to model with self-attention in the pre-training phase, which improves generalization ability. |
Sigmoid Head for Quality Estimation under Language Ambiguity (2026.acl-long)
Copied to clipboard
| Challenge: | Language model (LM) probability is not reliable quality estimator, as natural language is ambiguous. |
| Approach: | They propose to train a language model (LM) probability module on top of pre-trained LMs to address these limitations. |
| Outcome: | The proposed module is an extra unembedding head with sigmoid activation to tackle the first limitation. |
“A Little is Enough”: Few-Shot Quality Estimation based Corpus Filtering improves Machine Translation (2023.findings-acl)
Copied to clipboard
| Challenge: | Quality Estimation (QE) is the task of evaluating the quality of a translation when reference translation is unavailable. |
| Approach: | They propose a Quality Estimation based Filtering approach to extract high-quality parallel data from the pseudo-parallel corpus. |
| Outcome: | The proposed approach improves the machine translation system performance by up to 1.8 BLEU points over the baseline model. |
FairQE: Multi-Agent Framework for Mitigating Gender Bias in Translation Quality Estimation (2026.acl-long)
Copied to clipboard
| Challenge: | Existing QE models exhibit systematic gender bias, especially in gender-ambiguous contexts. |
| Approach: | They propose a multi-agent-based, fairness-aware QE framework that mitigates gender bias in both gender-ambiguous and gender-explicit scenarios. |
| Outcome: | The proposed framework mitigates gender bias in gender-ambiguous and gender-explicit scenarios while maintaining the strengths of existing models. |